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Probability Theory and 
Stochastic ocesses 

A basic tool designing of digital 
communication systems including 

□ Modeling sources that generate the information. 

□ Digitization of the source output. 

□ Characterizing of various communication channels. 

□ Designing of the receiver structure. 

□ Evaluation the performance of the whole system. 


Probability 

Theory 


Concepts of Probability 

Deals with averages of mass phenomena 
occurring sequentially or simultaneously 

> Electron emission, 

> Telephone calls, 

> Radar detection, 

Quality control, 

System failure, 

> Birth and death rates, 

The purpose is to predict such averages 
in terms o f probabilities of events. 


Probability oi an Event 

If an experiment is performed N times 
and the event A occurs N a times, then the 
probability of A is: 

m = £ 

> N a : Number of outcomes in favor to event A. 

> N : Number of all possible outcomes. 


Inion and In terse Limn 


Union A +B of two events is the event that 
occurs when A ox Box both occur: 


Intersection, or joint A# is the event that 
occurs when both events A and B occur. 


□ If A and B interleaved each other, a union 
and joint probabilities are interrelated by: 

P(A + B) = P(A) + P(B) - P(A . B) 

□ If A and 3 are independent, a union and joint 
probabilities are unrelated so that: 

P(A+B)=P(A)+P(B) 


Conditions] Probability 

If outcomes of two events A and B form 
their joint A the conditional probability of 
event A given event B is defined as: 

P(A* B) 

P(B) 

If events A and B are independent, then there is 
no joint occurrence, so that: 

P(A/B) = P(A) 

So, joint of two independent events equals the 
product of their individual probabilities: 

P(A . 8) = P(A). P(B) 



Binomial Distribution 


Assume p is the probability of occurrence of 
an event A. So, probability of non-occurrence 
o l such event will be termed as q = 1 - p % 

After repeating the experiment N times, the 
probability that A is observed exactly k times 
out of N trials, is a Binomial distribution: 
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Binomial Ap p I i ;atio:_s 

It involves any experiment for which there 
are only two possible outcomes on any trial. 



□ Passing or failing an exam, 

□ Receiving "O' or " V in a digital bit stream 
transmission, and 


J Occurrence or non-occurrence o any event. 





. andom Variables 


A random variable is a number assigned to 
every outcome of an experiment. It could be: 

> Voltage of a random source, 

> Phase of a random signal, 

> Power of a received signal, or any other. 

It is characterized by three basic functions 

□ Cumulative distribution function cdf. 

□ Probability density function pdf. 

□ Mass function. 


Cumulative Distribution 

function 

Probability that: Random variable X takes 
on a value equal to or less than x in a trail. 

F* Or) = P(X < x ) 

The discrete random variable by flipping a 
fair coin has cdf in Figla, with 2 steps. 

lie random variable by tossing a fair die in 
Figlb of 6 jumps, one at x = 1,2, . . ., 6. 


Examples on 

Cumulative Distribution Functions 


There are two jumps in F x (x), 
o One atx = -1 and 


o One at x = 1 . 
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(a) Tossing a Coin 


(b) Tossing a Die 


Fig. 1.1 Examples of Discrete CDF 



0 < F x (x) < 1 


Fx( x l) — Fx(%2)> if *1 - 

P(%\ < X < X 2 ) — F x (Xi) ~ F x (x\) 



i i obabilit / Density 

I ’unct I on 

pdf is the derivative o i the cumulative 
distribution function cdf: 

So, cdf, can be given in terms of the density 
function, pdf, by integration: 






Concerned with discrete variables as: 

PxM = P(X = x) 


For random variable of discrete values 

Pi = P(x = x t ) 


pdf can be evaluated in terms of the 
mass functions as follows: 



Two Random 
Variables 


wo Ra idon> Vi riabJf, 

Over finite interval x t <X < x 2 > and y t <Y < y 2 : 

I I 


So. cdf is: 
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For x quite independent on y: 
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Independent 


i andom Va iabl es 


If variables X and Y, are independent: 


P(x < X < x + dx, y < Y < y + dy) = [f x (x)dx] [f y (y)dy] 


P(x i < X < x 2 ,y t < Y < y 2 ) = 


" r x 2 " ry 2 

f x (x)dx f y (y)dy 

Jx ± J Jyx 


fxyix.y ) = f x (x)fy(y) 



tatistical Averages 



Average value is the expectation of the random 
variable X, and is given by: 


X = £(X] = m 
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For a continuous random variable: 

X = £[X] | - wt » I at /(at) dar 
For a function g(x) of the variable X is: 

&(x) = EtoW] = I s(x) /(*) *e 
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If the random variable X is raised to a 
power /?, the average value of X n is 
referred to as the n lh moment of the 
random variable X given by: 

T* = E\ *:*] = J 




Variance a 2 is a measure of the width of 
the probability density function: 


Is equivalent to the average value of the 
second moment (X-m ) 2 as: 

a 2 = E[(X - m) 2 ] = - m ) 2 /(x) dx 

o 2 — E[X 2 — 2 mX + m 2 ] 

= £[X 2 ] - 2 mE[X] + m 2 = E[X 2 ] - 2 m 2 + m 2 
= E[X 2 ]-rn 2 =E[X 2 ] ifm = 0 


Gaussian Distribution 


Many natural events are characterized by 
Gaussian density such as thermal noise. : 
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Error Function 


Cumulative function cdf of Gaussian density 
f(x) with zero mean is: 





This integral is not easy and is available in 
tables and is termed error function written as: 


erf vt 
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erf(O) = 0, erf(oo) = 1 



mplementary Error 

Fui tio 


Complementary error function erfc u defined: 
erf tu = 1- erfw = -= I r* du 

One could prove that cumulative distribution 
of Gaussian process is given as: 
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Densit / tfy = g (x) 

□ Given a random variable X 
where its associated density 
function is f x (x) 

□ What is the density function 
of random variable Y = g(x) 
where g(x) is some function 
ofX? 


JeLSJiy oil' y = g(x) 

Solve the equation: 

Y = 9 00 * 

Assume it has x n real roots. 

The density function will be: 

f ^ . /it fan) 
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Examples of g(x) 

1) If g(x) is a linear function, Y = a X + b: 

o It has one solution x = (y — b/d) and 
o The derivative is g'(x) — a, hence: 

y-b' 
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2) If g(x) is the inverse, Y = 1/X: 

o It has one solution x — 1/y, 
o The derivative is g'{x) = — - corresponds to y 2 : 




f* Cxi ^ 





g ( x ) is Sinusoidal 

If g(x) is sinusoidal, Y = a sin(^f + 0) 

It has n solutions f x (x n ) 

The derivatives is: g'(x) = a cos(x + 0) 

♦** y 2 + [#'(x)] 2 = a 2 sin 2 (x + 0) + a 2 cos 2 (x + 0) = a 2 

a W(x)Y= (Y - y 2 or a g'(x) = yja 2 -y 2 

Hence, the density function of Y is given by: 
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Central Limit Theory 

Density of sum of N independent Gaussian random 
variables approach Gaussian density as N increases. 

I Its mean is the same mean of N independent random 
variables, whereas standard deviation is ff/VN 

This theorem is even applied when individual 
random variables are not Gaussian. 

It is also applies in certain special cases even when 
individual random variables are not independent. 




Random Process 

A random process X (A, t ) is a function of two variables: 
an event A and time t. 

Fig illustrates random process with N sample functions of 
time {Xj (t)}each as the ou put of different noise generator. 

□ For a specific event Aj* we have single time function 
X(Aj, t ) = Xj (t) (i.e. single sample function). 

□ For a specific time t k ,X(A,t k ) is a random variable 
X(t k ) whose value depends on the events, 

□ For specific event Aj and specific time t = t k % 
X(Aj, t fe ) is simply a number. 

The totality of all sample functions is called an ensemble. 


Real number , x 





Statistical Properties 

A random process whose distribution functions are 
continuous can be described statistically with (pdf), 

> But generally, pdf of a random process will be 
different for different times. 

> Also in most cases it is not practical to get empirically 
the probability distribution of a random process. 

However, a partial description consisting of the mean 
and autocorrelation function are often adequate for the 
needs of communication systems. 

We define mean & autocorrelation of random process 
X (t) as: 


Mean & Autocorrelation 


We define the mean of the random process X (t) as: 



X(t k ); the random variable obtained by observing the 
random process at time t k 

Px k (x); pdf of X(t k ) over the ensemble of events at t k . 

Autocorrelation of random process X ( t) is a function of 
two times t* and t 2 : (measure of the degree to which 2 
time samples of the same random process are related) 


R x (.h,t 2 ) = E[X(t 1 )X(t 2 )] 


Xfti) and^(t 2 ); are the random variables obtained by 
observing X (t) at times and t 2 . 


Stationary 




Process 


Random process X (t) is said to be strict-sense stationary 
if none of its statistics are affected by a shift in time. 


And, is said to be wide-sense stationary (WSS) if two of 
its statistics (mean and autocorrelation) do not vary with a 
shift in time. Thus, a process is WSS if 

E [X (t)] = m x = constant 


Rx(ti> * 2 ) — Rx(ti ~ * 2 ) 


Strict-sense implies wide-sense, but not vice versa. 

Most results in communication theory are predicated on 
random signals and noise being wide-sense stationary. 

In practice, it is not necessary for a random process to be 
stationary for all time but only for interval of interest. 


Autoco relation of WSS 


Variance; a measure of randomness for random variables. 
Autocorrelation; a measure for random processes. 

For WSS process, autocorrelation is a function of time 
difference t = — 1 2 ; 

R x (x) = E[X(t)X(t + t)] for — oo < t < oo 

For zero mean WSS process, Rx(r) indicates the extent to 
which random values of process separated by x seconds 
are statistically correlated. 

It gives an idea of frequency response that is associated 
with a random process. 


Autocorrelation and Response 

It gives an idea of frequency response that is associated 
with a random process. 

If it changes slowly as x increases, it indicates that, sample 
values of X ( t ) taken at t t and + r are nearly the same. 
Thus, we would expect a frequency domain of X ( t) to 
contain low frequencies. 

Or, if it decreases rapidly as r is increased, X ( t ) expected 
to change rapidly with time and thereby contain mostly 
high frequencies. 

Properties of autocorrelation of a real-valued wide-sense 
stationary process are in Table. 1.1. 


Table. 1.1: Properties of Autocorrelation of a 
Real -valued Wide-sense Stationary Process 

n 

Property 

Meaning 


Rx(t) = Rx(-t) 

Symmetrical in x about zero 

2 

RxCO — Rx(®) 3ll T 

maximum value occurs at the 
origin 

3 

Rx(x) ** G x (f) 

Autocorrelation and power 
spectral density form a Fourier 
transform pair 

1 

R x (o) = E{x 2 (t)} 

Value at the origin is equal to 
the average power of the signal 








Time & Ensemble Average 

With ensemble averaging, we have to average across all 
the sample functions of the process and would need to 
have complete knowledge of the first and second-order 
joint pdfs. Such knowledge is generally not available. 

When random process is ergodic, its time averages equal 
to its ensemble averages, and the statistical properties of 
process can be determined by time averaging over a single 
sample function of the process. 

To be ergodic, it must be stationary in strict sense. 

Ergodic process is stationary, but, stationary process is not 
necessarily ergodic. 


Ergodic Pro jess 

For most communication, we are satisfied to meet 
conditions of wide-sense stationary, we are interested 
only in the mean and autocorrelation. 


Random process is ergodic in mean and autocorrelation if: 
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X(t) X(i + t)A 


- 7/2 


Reasonable assumption in most communication signals 
(in absence of transient) is that the random waveforms are 
ergodic in the mean and the autocorrelation function. 



i)en>ity 



Powe r Spectral Density 

Power spectral density psd is very useful in 
communication, it describes distribution of a 
signal ’s power in the frequency domain. 

It enables us to evaluate the signal power that 
will pass through a network having known 
frequency characteristics. 

We summarize the principal features of psd 
as follows: 


Features of psd 

□ G x (f ) > 0 and is always real valued 

□ G x {f ) = G x (— /) for ^ (t) real valued 

Q Gx(f) ** ^jr( T ) 

□ Relationship between average normalized 
power and psd: 

~ Gx(f) df 




ise 

□ Define: Unwanted signals that disturb transmissi on 
and processing of signals in communication systems. 

□ T^pes: There are many sources of no ise: 

1 - External: atmospheric, galactic, man-made noise 

2- Internal to the system: 

Shot noise, arises because of the discrete nature of 
current flow in electronic devices. 

Thermal noise, which is due to the random motion of 
electrons in a conductor. 

□ Analysis: Analysis is usually based on a source of 
noise called white-noise. 



* Stationary Noise 


White noise, denoted by W (t), is a stationary process 
whose power spectral density S w (/) has a constant 
value across the entire frequency interval. 



Autocorrelation R w (r) is the inverse Fourier transform 
of power spectral density ( Wiener-Khintchine): 



Since Rw CO is zero for r =£ 0, so any two different 
samples of white noise are uncorrelated no matter how 
closely together in time those two samples are taken. 




Power Spectral Density (b) Autocorrelation Function 

Characteristics of White Noise 


Characteristic of Noise 


White noise is also Gaussian, so the two samples are 
statistically independent. Thus, white Gaussian noise 
represents the ultimate in “randomness". 

Utility of white-noise process in the noise analysis of 
communication systems is parallel to the use of delta 
function in the analysis of linear systems. 

As long as bandwidth of noise process at the input of a 
system is larger than the bandwidth of the system itself, 
then we may model noise process as white noise. 


IdJ sal Low 



Filter &d 


Noise 


Ideal LP Filtered Noise 

If white Gaussian noise of zero mean and N 0 /2 power 
spectral density is applied to ideal LPF of band 5, output 
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Autocorrelation function is inverse Fourier transform: 

f WL 

#M?) = I = 

So, If input noise is Gaussian, band-limited noise N(t) 
at filter output is also Gaussian. 

If N(t) is sampled at 25, resulting noise samples are 
uncorrelated, Gaussian, and statistically independent. 


Id aal LP Filter 


Noise 

AWGN 




(a) Power Spectral Density (b) Autocorrelation Function 

Characteristics of Low Pass Filtered White Noise 


Noise 
Correlation 
with Sinusoid 


Correlation of Noise with sinusiod 


Consider output of correlator with white Gaussian noise 
w(t) and sinusoidal wave at its two inputs; 
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If w(t) of zero mean, output w'(i) has zero mean too. 
Variance of correlator output is therefore defined by: 
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Assume the frequency f c of sinusoid is integer multiple 
of the reciprocal of T for mathematical convenience. 


Correlator' 


Noise 

AWGN 
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Narrowband 




Narowbandi i Joise 

Receiver includes preprocessing for received signal. It 
takes a form of narrowband filter of enough band to pass 
modulated component of received signal undistorted, to 
limit the effect of channel noise passing in the receiver. 

The noise process appearing at the output of such a filter 
is called narrowband noise. 

If spectral components of narrowband noise centered 
about midband frequency ±£, we find that a sample 
function n(t) of such a process appears somewhat 
similar to a sine wave of frequency f c . 

The sample function n(t) may, then, change slowly in 
both amplitude and phase, as in Fig. 




(a) Power Spectral Density (b) Sample Function 

Narrowband Noise 



Narowband i Joise 

Consider n(t) produced at narrowband filter output in 
response to the sample w(t) of white Gaussian noise 
process of zero mean and unit power spectral density 
of the respective processes N(t) and W (t). 

We express power spectral density of A/(t) in terms of 
the transfer function o f filter H(f) as: 

S N (/) = [H(f)] 2 

So, any narrowband noise encountered in practice may 
be modeled by applying a white-noise to a suitable 
filter in the manner described as above. 


In 




anents 


Inphase and Quadrature 

C omponents ol r loise 

Often ly, it is convenient to represent the 
narrowband noise n(t) in terms of its in- 
phase component n t (t) and quadrature 
component Uq (t) as follows: 

n(t) = n,(t) cos(2ir/ c ) - n Q (t) sin(2jr/ c ) 

So, we have the following properties 


Proper ties 

First: In-phase n t (t) and quadrature Uq (t) narrowband 
noise n(t) have zero mean. 

Second: If narrowband noise n(t) is Gaussian, its in- 
phase and quadrature components are jointly Gaussian. 

Third: If narrowband noise is weakly stationary, its in- 
phase and quadrature are jointly weakly stationary. 

Forth: Both in-phase and quadrature noise have same 
power spectral density, which is related to the power 
spectral density S NN (f ) of the original narrowband noise 


Continue Properties 

Fifth: In-phase and quadrature components have the 
same variance as the narrowband noise n(t). 

Six: Cross-spectral densities of in-phase & quadrature 
components of a narrowband noise are pure imaginary: 

Seventh: If narrowband noise n(t ) is Gaussian with 
zero mean and S N (/ ) power spectral density, that is 
locally symmetric about the midband frequency ±/ c , 
the in-phase and quadrature noise are statistically 
independent. 


